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Abstract 


This report details the development and validation of the ENGAGE Grades 6-9, a 
measure of academic behavior designed to determine students’ levels of academic risk. The 
work presented in this report is part of a comprehensive research program of educational risk 
assessment based on key academic behavior predictors (also known in the literature as 
socioemotional or psychosocial factors) of academic success and persistence. ACT researchers 
modeled early high school GPA based on a prospective sample of 4,660 middle school students 
from 24 schools and 13 districts. The findings show that (a) prior grades and standardized 
achievement scores are the strongest predictors of high school academic success, and (b) 
academic behavior (e.g., motivation, social engagement, and self-regulation) adds substantial 
incremental validity to the prediction of success. These findings highlight the importance of 
effective risk assessment based on a combination of achievement and behavior factors amenable 
to intervention. The discussion focuses on how educators can use such information to identify at- 
risk students early, assess their strengths and needs, and provide them with interventions 
designed to facilitate academic success. 
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Development and Validation of ENGAGE Grades 6-9 


High academic failure and dropout rates remain significant issues in the United States, 
with estimates of over 25 percent of public school students failing to earn a high school diploma 
(Education Week, 2009; Stillwell, 2009). In some states and communities, these rates exceed 50 
percent of all entering 9 th -grade students. The No Child Left Behind (NCLB) legislation includes 
systematic monitoring of student progress through standardized achievement testing, but this 
alone does not ensure the proper identification of and intervention with at-risk students. We now 
know that measuring critical noncognitive factors can increase schools’ abilities to identify and 
intervene with students at risk of academic failure and dropout (ACT, 2008; Zins, Bloodworth, 
Weissberg, & Walberg, 2004). In the psychological literature, noncognitive factors are often 
referred to as socioemotional learning (SEL) factors, psychosocial factors (PSFs), and behavioral 
factors. These tenns are often used interchangeably to describe “academically and 
occupationally relevant skills and traits that are not specifically intellectual or analytical in 
nature” (Rosen, Glennie, Dalton, Lennon, & Bozick, 2010). Noncognitive factors include a 
range of attitudinal, behavioral, emotional, and personality characteristics that facilitate 
functioning well in one’s environment, including school and work. It is worth noting that some 
of these factors tend to be more visible to others (e.g., behaviors like completing homework or 
certain personality traits like extraversion), whereas others are less visible (e.g., attitudes about 
school, emotions, and the processes by which individuals regulate them). Since this report is 
geared toward an education audience, we will use the tenn “academic behavior” as an umbrella 
that encompasses the full range of noncognitive factors (i.e., attitudinal, behavioral, emotional, 
and personality). 

After reviewing studies of student dropout conducted over the past 25 years, Rumberger 
and Lim (2008) identified several factors that differentiate students who graduate from those 
who drop out of high school. Their list includes learning behaviors, attitudes, demographics, and 
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characteristics of family and school. Similarly, the Baltimore Education Research Consortium 
(Mac Iver, 2010) found that poor grades and course failure is a strong predictor of high school 
dropout. These findings, among others, point to the importance of including psychosocial and 
behavioral data when assessing academic risk. 

Several single-sample studies have examined the direct and indirect effects of 
noncognitive factors on academic success and retention rates, highlighting a range of constructs, 
including self-efficacy, motivation, locus of control, attitude toward learning, attention, and 
persistence, as well as strategy and flexibility (Grigorenko, Jarvin, Diffley III, Goodyear, 
Shanahan, & Sternberg., 2009; Yen, Konold, & McDemott, 2004). These findings are consistent 
with results of a large-scale review of student success research conducted in elementary and 
middle school, where a range of noncognitive constructs — self- and social-awareness, self- 
management, relationship skills, and responsible decision-making — were found to be key 
antecedents of student academic perfonnance (e.g., Payton, Weissberg, Durlak, Dymnicki, 
Taylor, Shellinger, & Pachan, 2008). 

Analogous results have been obtained during the transition to postsecondary education, 
where a combination of noncognitive factors, academic performance, and standardized 
achievement factors are highly predictive of first-year college academic success and retention 
behavior (Robbins, Allen, Casillas, Peterson, & Le, 2006). Using an assessment model based on 
motivation, social engagement, and self-regulation (see Le, Casillas, Robbins, & Langley, 2005; 
Robbins, Lauver, Le, Davis, Langley, & Carlstrom, 2004), noncognitive factors made 
incremental contributions beyond high school grade point average and standardized achievement 
for predicting college academic performance and persistence behavior (Robbins et al., 2006). 
Similar findings also have been obtained when predicting graduate school outcomes, where 
personality and behavioral data provide incremental validity contributions after controlling for 
college GPA and standardized test scores (Kyllonen, Walters, & Kaufman, 2005). 
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What is ENGAGE Grades 6-9? 

Research suggests that one of the most effective ways to prevent poor academic 
performance and student dropout is to identify at-risk students early and assist them in their 
educational development (Balfanz, Herzog, & Mac Iver, 2007). Although standardized 
achievement tests have been shown to be valid and useful measures for predicting academic 
outcomes (ACT, 2007; Willingham, Lewis, Morgan, & Ramist, 1990), the utility of these 
measures for predicting student success and identifying at-risk students can be improved with 
assessment of academic behaviors. 

ENGAGE Grades 6-9 (formerly known as the Student Readiness Inventory — Middle 
School) is designed to identify youth who are at academic risk by augmenting standardized 
achievement testing with measures of important academic behaviors. It is a low-stakes, self- 
report inventory made up of ten scales (106 items, 4 th -grade reading level) that can be generally 
organized into three broad domains shown to be predictive of academic performance and 
persistence (Robbins et ah, 2004, 2006; Robbins, Oh, Le, & Button, 2009): 

■ Motivation includes personal characteristics that help students to succeed academically 
by focusing and maintaining energies on goal-directed activities. 

■ Social engagement includes interpersonal factors that influence students’ successful 
integration into their environment. 

■ Self-regulation includes the thinking processes and emotional responses of students that 
govern how well they monitor, regulate, and control their behavior related to school and 
learning. 

ENGAGE captures students’ perceptions of themselves, their families’ commitment to 
education, school-related factors, and important behavioral indicators. It is designed for students 
in grades 6 through 9. Results from ENGAGE are reported as percentile ra nk scores based on 
norms from the ENGAGE Grades 6-9 field study (ACT, 2011a) and indicate whether students 
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have developed the academic behaviors necessary to succeed in high school. In addition to 
individual student and advisor reports, both of which provide a broad profde of students’ 
strengths and needs across the ten scales measured by ENGAGE, the instrument provides an 
academic success index (included on the advisor report only). This index is based on ACT’s 
research using ENGAGE scales along with other information reported by students (e.g., previous 
grades, homework completion) to estimate the probability that a student will be academically 
successful (defined as obtaining a high school GPA of 2.0 or above). Further, ENGAGE 
provides a roster report, which allows educators to sort, filter, and merge student scores, and it 
also provides school and district aggregate reports, which allow school administrators to review 
the academic behaviors of entire groups of students (either at the school or district level). Based 
on the student-level information provided by ENGAGE, educators can identify students who 
may be at-risk of experiencing academic difficulties and connect them to interventions based on 
their areas of need. Further, administrators can use the aggregate-level data to assess whether 
group-based interventions (in a particular behavior area or at a particular school) may be needed. 
For more details on how to use ENGAGE and to view sample reports, consult the ENGAGE 
Grades 6-9 User’s Guide (ACT, 2011a). 

This report details the development and validation of ENGAGE, as well as its utility for 
identifying students who are at-risk of poor academic performance. The report concludes with a 
discussion of practical applications. 

Developing a Comprehensive Conceptual Model 

Development of ENGAGE followed a construct validation approach (Clark & Watson, 
1995; Loevinger, 1957; Nunnally & Bernstein, 1994), a multi-step process by which researchers 
(a) describe a theoretical model, referred to as the “nomological net” (Cronbach & Meehl, 1955) 
consisting of one or more hypothetical constructs and their relations with one another and to 
observable criteria, (b) build measures identified by the theory, and (c) empirically test the 
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hypothesized relations between the constructs and observable criteria. (For a detailed review of 
the construct validity approach to test construction, see Simms & Watson, 2007.) 

The process for developing ENGAGE began with a thorough literature review to identify 
predictors of student academic perfonnance and persistence in high school. Based on the 
existing literature, ACT researchers generated a comprehensive conceptual model for assessing 
middle school academic risk. Developers focused on predictors from five primary categories: 

1) prior academic achievement, 

2) noncognitive factors including motivation, social engagement, and self-regulatory 
factors, 

3) observable behavioral indicators including time spent on homework and 
absenteeism, 

4) school factors including average class size and percent of students eligible for 
free/reduced lunch, and 

5) demographic factors including gender, race/ethnicity, and parental education. 

This model is detailed in Table 1 (see on following page). 

Our goal was to assess the relative importance of each category of variables across a 
large number of independent studies. We relied on effect size, a measure of the strength of the 
relationship between two variables, to summarize the results. We focused on bivariate 
relationships. It is worth noting that when these variables are combined into a single 
model/study (as in ENGAGE) each will likely make a smaller contribution because of 
intercorrelations. Thus, only articles and chapters containing sufficient information to calculate 
an effect size (e.g., means and standard deviations or correlations) were included. Key results 
for the outcomes of persistence and academic achievement are summarized below. 
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TABLE 1 

Predictors Included in the Model of Middle School Academic Risk 


Categories 

Predictors 



Academic 

Achievement 

■ school grades 

■ EXPLORE scores 



Psychosocial 

Factors 

Motivation 

■ Academic 
Discipline 

■ Commitment to 
School 

■ Optimism 

Social Engagement 

■ Family Attitude toward 
Education 

■ Family Involvement 

■ Relationships w/School 
Personnel 

■ School Safety Climate 

Self-Regulation 

■ Managing Feelings 

■ Orderly Conduct 

■ Thinking Before 
Acting 

Behavioral 

Indicators 

■ days absent 

■ homework not done 

■ media time 

■ # of times changed schools 

■ time spent on homework 


School 

Factors 

■ percent free/ 
reduced lunch 

■ percent minority 

■ average class size 


Demographic 

Factors 

■ gender 

■ race/ethnicity 

■ parental education 

■ home resources 

■ family education aspirations 


Student Academic Achievement 

As predictors of dropout, general academic competence and overall high school GPA 
varied considerably in terms of effect size, with most of them being small (range d = .14 to .28) 
(Cairns, Caims, & Neckersman, 1989). However, specific core academic competencies such as 
English, math, and reading had higher effect sizes, ranging from moderate to large (range d = .44 
to .80) (Fitzsimmons, Cheever, Leonard, & Macunovich, 1969; Kaufman & Bradbury, 1992; 
Lloyd, 1978). Having been held back in school consistently showed large effects (mean d = .96). 
This variable was also a good predictor of subsequent academic achievement, with a moderate to 
large effect size (range d= .39 to 1.00) (Lloyd, 1978; Rumberger, 1995). 
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Psychosocial Factors 

The literature shows that student personality characteristics, student attitudes, and 
parental/family attitudes are all relevant predictors. Each of these is summarized below. 

Student personality. Several personality characteristics have been found to predict both 
dropout and achievement, with a broad range of effect sizes reported. For example, motivation 
constructs, such as achievement motivation, academic discipline, as well as disinhibition and 
impulsivity, have demonstrated moderate effect sizes (mean d = .43) (Lounsbury, Saudargas, & 
Gibson, 2004; Peterson, Casillas, & Robbins, 2006; Robbins et ah, 2006; Watson & Clark, 
1993). Emotional stability and related constructs, such as aggression, self-esteem, and academic 
self-confidence demonstrated small to moderate effect sizes (mean d = .36) (Robbins et ah, 2006; 
Marsh & Yeung, 1997). Further, optimism and self-efficacy showed moderate to large effect 
sizes (range d = .50 to .75) (Lounsbury et ah, 2004; Capella & Weinstein, 2001; Skinner, 
Wellborn, & Connell, 1990). 

Student attitudes. As predictors of dropout, measures of students’ general attitude toward 
school showed small effect sizes (mean d = .26) (Alexander, Entwisle, & Kabbani, 2001), 
whereas measures of school commitment had a moderate predictive effect (mean d = .52) 
(Robbins et ah, 2006; Vallerand, Fortier, & Guay, 1997). Students’ expectations of educational 
attainment also had a small to moderate effect (mean d = .31) (Rumberger & Larson , 1998). As 
predictors of academic achievement, both general attitude and commitment to school showed 
moderate to large effects (range d = .29 to .80) (Marchant, Paulson, & Rothlisberg, 2001; 
Peterson et ah, 2006). 

Parental/family attitudes. As predictors of dropout, general parent and family attitudes 
toward education showed small effect sizes (range d = .06 to .14) (Alexander et ah, 2001). 
However, more specific measures of parental attitudes toward education (e.g., communication 
with students about school, parental involvement in school activities, and educational 



expectations for students), evidenced large effects (range d = .95 to 1.67) (Jimerson, Egeland, 
Sroufe, & Carlson, 2000; Kaufman & Bradbury, 1992). These measures also were moderate to 
strong predictors of achievement (range d = .39 to .90) (Jimerson et ah, 2000; Marchant et al., 
2001; Seyfried & Chung, 2002). 

Behavioral Indicators 

As a predictor of dropout, disorderly conduct showed moderate effects (mean d = .50) 
(Finn & Rock, 1997; Kaufman & Bradbury, 1992). Further, student absenteeism showed 
moderate to large effects (mean d = .70) (Finn & Rock, 1997; Rumberger, 1995; Worrel & Hale, 
2001). Time spent doing homework varied in effect size from small to large (range d = .22 to 
1.30), depending on the response options (and resulting variance) used in each study. Generally, 
a greater range of response options led to higher effects (Kaufman & Bradbury, 1992). As a 
predictor of academic achievement, engaging in disruptive behavior showed moderate to large 
effects (range d = .40 to .80) (Kaufman & Bradbury, 1992), and student absenteeism evidenced 
moderate to very large effects (range d = .50 to 2.00) (Kaufman & Bradbury, 1992; Rumberger, 
1995; Worrell & Hale, 2001). 

School Factors 

As predictors of dropout, a number of school characteristics (e.g., mean achievement 
scores, size, percent of minority students, percent of students eligible for free/reduced lunch) 
showed moderate effects (mean d = .65) (Kaufman & Bradbury, 1992), whereas measures of 
school climate showed small effects (mean d = .15). However, few studies have included 
measures of safety, which were reported to be predictive in some qualitative reports (e.g., 
Bridgeland, Dilulio, & Morison, 2006; Skiba, Simmons, Peterson, & Forde, 2006). As 
predictors of student academic achievement (e.g., grades), both school climate and mean 
standardized achievement scores (at the school level) showed moderate effects (mean d = .51) 
(Kaufman & Bradbury, 1992; Marchant et al., 2001; Worrell & Hale, 2001). 



9 


Demographic Factors 

The research on demographic variables has been primarily used to examine dropout. In 
this context, age, ethnicity, and socioeconomic status (SES) showed moderate to strong effects 
(average d = .85, .46, and .64, respectively) (Cairns et ah, 1989; Kaufman & Bradbury, 1992; 
Rumberger, 1995). Further, mobility (i.e., number of school moves) evidenced small to large 
effects, with larger effects for greater frequency of moves (range <7 = . 1 4 to 1.21) (Kaufman & 
Bradbury, 1992; Rumberger, 1995; Rumberger & Thomas, 2000). 

Theoretical Context 

The aforementioned findings fit well within the broader theoretical literature on 
psychosocial predictors of academic success and persistence, in particular motivation, social 
engagement, and self-regulation (cf. Robbins et al., 2004). For example, Kanfer and Heggestad 
(1997, 1999) illustrated the importance of motivation and self-regulation as proximal 
detenninants of goal-oriented behaviors when examining the effects on learning. Motivation and 
self-regulation are undergirded by three critical processes: self-monitoring, self-evaluation, and 
self-reactions (Kanfer & Ackennan, 1989; Kanfer & Heggestad, 1997). These processes are 
malleable to change through targeted intervention and reflect students’ ability to motivate 
themselves to achieve via self-regulatory processes (Kuhl, 1985) as observed in study habits and 
skills. Robbins et al. (2009) included social engagement as another key mechanism in student 
success, especially as it relates to persistence behavior. Social engagement is associated with 
college retention (Robbins et al., 2006; Robbins et al., 2009) and could also be a mechanism to 
prevent high school dropout behaviors. Helping students feel connected to their school 
environment and able to take advantage of family, peer, and school supports can promote 
feelings of “belonging” in an educational environment, which is necessary for continued 
persistence behavior. 
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Inventory Development and Model Refinement 

The conceptual model detailed in Table 1 provided the foundation for the ENGAGE scale 
development process, particularly for measures of student psychosocial characteristics and 
behavioral indicators. Briefly, ACT researchers developed items targeting the constructs in the 
conceptual model. These initial items were administered to samples of middle school students 
and exploratory and confirmatory factor analyses were used to examine the factors underlying 
these data. Items were screened based on their loadings on the factors. After collecting data from 
additional student samples, ACT researchers have continued to track these students throughout 
high school to collect outcome data and refine prediction models. Details about each of these 
steps are provided below. 

Item Writing 

A team of four applied psychologists wrote items to represent each of the psychosocial 
constructs included in the conceptual model. (See Table 2 for scales and definitions.) For each 
construct, the writers developed a definition and then wrote items to capture the construct. The 
writers first generated items independently and then met to discuss the items to be retained 


and/or revised. 
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TABLE 2 

ENGAGE Grades 6-9 Scale Definitions, Sample Items, and Behavioral Indicators 1 


Domain 

Scale Name 

Definition 

Sample Item 

Motivation 

Academic 

Discipline 

Degree to which a student is 
hardworking and conscientious as 
evidenced by the amount of effort 
invested into completing schoolwork. 

I turn in my 
assignments on time. 


Commitment to 
School 

Commitment to stay in school and 
obtain a high school diploma. 

I am committed to 
graduating from high 
school. 


Family Attitude 
toward Education 

Positive family attitude regarding the 
value of education. 

My family supports my 
efforts in school. 

Social 

Family 

Involvement 

Family involvement in a student’s 
school life and activities. 

I talk to my family 
about schoolwork. 

Control 

Managing Feelings 

Tendency to manage duration and 
intensity of negative feelings and to 
find appropriate ways to express 
feelings. 

I would walk away if 
someone wanted to 
fight me. 


Optimism 

A hopeful outlook about the future in 
spite of difficulties or challenges. 

I am confident that 
everything will turn out 
all right. 


Orderly Conduct 

Tendency to behave appropriately in 
class and avoid disciplinary action. 

I have been sent to the 
principal’s office for 
misbehaving. 


1 Based on research concerning the optimal number of response options in Likert-type scales (Green & Rao, 1970; 
Matell & Jacoby, 1972), items were set to a 6-point Likert-type response scale, ranging from strongly disagree (1) to 
strongly agree (6). The Orderly Conduct items, which use a Yes/No response scale, were an exception. 


12 


TABLE 2 (continued) 


Domain 

Scale Name 

Definition 

Sample Item 

Self- 

Regulation 

Relationships with 
School Personnel 

The extent to which students relate to 
school personnel as part of their 
connection to school. 

Adults at my school 
understand my point of 
view. 


School Safety 
Climate 

School qualities related to students’ 
perception of security at school. 

I feel safe at school. 


Thinking Before 
Acting 

Tendency to think about the 
consequences of one’s actions before 
acting. 

I think about what 
might happen before I 
act. 

Behavioral 

Indicators 

Absenteeism 

Number of absences, days tardy, and 
skipped classes reported by the 
student over the past month. 

How many days were 
you absent from school 
in the past month? 


Held back 

Having been held back from normal 
grade progress. 

Have you ever been 
held back from moving 
to the next grade? 


Homework time 

Time spent on homework on a typical 
school day. 

How many hours do 
you usually spend 
doing homework on a 
school day? 


Media time 

Time spent watching TV, playing 
video games, and surfing the Internet 
(for non-school related purposes) 
during a typical school day. 

How many hours do 
you usually watch TV 
on a school day? 


School Mobility 

Number of times that the student 
changed schools since starting 
elementary school. 

How many times have 
you ever changed 
schools? 


Preliminary Test of Item Clarity 

To ensure that the items would be comprehensible to middle school students, we 
administered the items to a small group of 6th- to 8th-grade students (N = 25; 72% White; 52% 
female; mean age = 12.3 years, SD = 1.0, range 1 1 to 14 years). The students were asked to rate 
the extent to which they understood the meaning of the items using a 5-point Likert-type scale 
ranging from very easy to understand (1) to very difficult to understand (5). Based on these mean 
ratings of item clarity, we deleted or revised items. Subsequently, the revised items were 
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presented to a group of experts in education and communication, who were asked to comment on 
item clarity. The items were again revised based on this feedback. 

Pilot Test 

Next, we collected data for a pilot version of ENGAGE items at four middle schools. 
The pilot study data were used to establish the initial structure and psychometric properties of the 
scales, as well as to guide item revision for the next phase of research and development. 
Advisory Group 

Concurrent with the pilot test, ACT staff convened an advisory group composed of a 
broad range of experts in fields related to K-12 education administration, academic 
failure/dropout, and secondary school remediation/intervention. The advisory group provided 
feedback on the model of middle school academic risk, our choice of scales and behavioral 
indicators, and some preliminary use cases. The general consensus was that this model 
appropriately captured the constructs theoretically expected to predict high school academic 
success. 

Based on results from the pilot test and the recommendations of the advisory group, ACT 
researchers made additional modifications to ENGAGE scales in preparation for the field study. 

Item Selection and Scale Structure 

To identify the factors (scales) underlying ENGAGE items and to identify those items 
most strongly associated with each scale, ACT researchers conducted both exploratory and 
confirmatory factor analyses using the pilot test sample (N= 1,664). 

Exploratory Factor Analysis 

We used exploratory factor analysis to understand the structure of ENGAGE item-level 
data and to select items. These analyses were carried out on two-thirds of the total sample from 
the pilot (N = 1,098). We used a “bottom-up” strategy in which we built scales independent of 
one another. That is, we ran a separate factor analysis for the items written to represent each 
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scale to determine whether they tapped a single dimension (as intended). We specified principal 
axis factoring as the extraction method and used varimax rotation (cf. Nunnally & Bernstein, 
1994; Gorsuch, 1997). In the majority of cases, a single factor solution was clearly indicated. In 
these cases, items with loadings of .40 or higher on the first principal component were retained 
for further analysis. 

Confirmatory Factor Analysis 

Next, we used confirmatory factor analysis to (a) confirm each of the dimensions 
identified in the previous step and (b) select a final set of items to represent each of the scales. 
The confirmatory factor analysis was carried out on the remaining one third of the total sample 
from the pilot (N = 566) using EQS software for structural equation modeling (Bender, 1995). 
For each dimension, we specified one-factor solutions using the items that were retained from 
the exploratory step. The fit of the intended model to the data was assessed using several fit 
indexes [i.e., Comparative Fit Index (CFI), Global Fit Index (GFI), Root Mean Square Error of 
Approximation (RMSEA), and Standard Root Mean Square Residual (SRMR); see Hu & 
Bender, 1999]. Items that showed clear loadings on their respective dimensions in the results of 
both the exploratory and confirmatory factor analyses were selected for the final versions of the 
scales. After the exploratory-confirmatory steps described above, approximately 30% of the 


items were discarded. 
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The Field Study 

Participants 

During the Spring and Fall of 2006, ACT staff recruited districts throughout the country 
to participate in the ENGAGE field study. To be invited to participate, schools needed to: (a) be 
made up of students representing a broad range of demographic and achievement characteristics, 
(b) use ACT’s EXPLORE program, and (c) agree to provide follow-up data as the students 
moved into and through high school. 

A total of 4,660 students from 24 middle schools from 13 districts participated in the field 
study. Most participating students spoke English as their primary language (98%) and were 
Caucasian (62%)'. Students’ mean age was 13.5 years (SD = .7), the sample was evenly split on 
gender (51% male), and the majority were enrolled in 8th grade (83%), with the remainder 
enrolled in 7th grade. 

Of the initial sample of 4,660 students, 3,757 (81%) also took the EXPLORE assessment, 
which is typically administered in the 8th grade. Among students who took EXPLORE, 51% 
were male, 98% spoke English as their primary language, and 64% were Caucasian. Students 
varied in age from 12 to 16 years old (M= 13.6, SD = .6). Thus, the subsample of students who 
took EXPLORE was similar in demographics to the larger study sample. 

As part of the study, school districts agreed to provide follow-up infonnation (i.e., 
enrollment status and high school grades) for the students who participated in the initial phase of 
the study. Follow-up infonnation was obtained two years later for 3,325 students enrolled in 9 th 
and 10 th grade at the time of follow-up, who comprised 71% of the original study sample and 
89% of the EXPLORE-tested sample. Demographic characteristics for the subsample of 
students with follow-up information were similar to those of the original sample (69% 

2 The remaining race/ethnicity breakdown was: 16% Hispanic/Latino, 10% African American, 3% Multiracial, 2% 
Asian American/Pacific Islander, 1% American Indian/ Alaskan Native, and 7% of “other” or unknown 
race/ethnicity. 
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Caucasian; 51% male; 99% spoke English as primary language; mean age =13.6 years, SD = .6). 
Academic behavior variables also were similar for the original sample and the subsample with 
follow-up data. 

Procedure 

ENGAGE was administered to the held test sample in group settings during class time. 
Students were told that participation was voluntary, their results would be kept confidential, and 
the questionnaire would require approximately 30 minutes. Participating schools and districts 
were provided with group-level summaries of their students’ results as an incentive for 
participating. Additional summaries were provided to districts after each follow-up wave of data 
collection. 

Psychometric Properties of ENGAGE Grades 6-9 

The following descriptive statistics, reliability estimates, and convergent/discriminant 
relationships with other scales and constructs were derived from the held study sample. 

Scale Reliability 

ENGAGE scales are relatively short (range = 9 to 12 items) and have good to excellent 
internal consistency reliabilities (Crobanch coefficient alpha range = .81 to .90; median = .87; 


see Table 3). 
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TABLE 3 

Descriptive Statistics and Intercorrelation Matrix for the ENGAGE Grades 6-9 Scales 


Scale Name (# of items) 

M 

SD 

1 

2 

3 

4 

5 

6 

7 8 9 10 

1 

Academic Discipline 
(11) 

4.80 

0.92 

.92 







2 

Commitment to School 
(10) 

5.63 

0.60 

.53 

.88 






3 

Optimism (10) 

4.79 

0.93 

.54 

.50 

.90 





4 

Family Attitude toward 
Education (10) 

5.55 

0.67 

.46 

.63 

.44 

.88 




5 

Family Involvement (9) 

4.67 

1.04 

.53 

.46 

.56 

.57 

.86 



6 

Relationships with Sch. 
Personnel (12) 

3.95 

1.02 

.47 

.35 

.54 

.33 

.53 

.90 


7 

School Safety Climate 
(11) 

4.19 

0.96 

.37 

.33 

.42 

.31 

.36 

.56 

.84 

8 

Managing Feelings (12) 

3.83 

1.20 

.53 

.29 

.40 

.25 

.38 

.48 

.37 .90 

9 

Orderly Conduct (9) a 

0.68 

0.30 

.55 

.29 

.32 

.25 

.32 

.38 

.34 .62 .83 

10 

Thinking before Acting 
(12) 

3.91 

1.10 

.51 

.28 

.41 

.25 

.37 

.43 

.31 .55 .49 .87 


Note. Range N = 2,982 to 4,646. Cronbach’s alphas featured on the diagonal in italics. 
Correlations with absolute value exceeding .05 are significant (p < .01). a Scale scored as Y/N; 
all other scales scored on a 6-pt Likert scale. 

Convergent / Discriminant Validity of ENGAGE Scales 

Intercorrelations among ENGAGE scale scores are presented to provide a clear picture as 

to how the scales of the instrument related to each other. Intercorrelations among the scales 

show a reasonable convergent/discriminant pattern, with scales generally correlating more 

strongly with those scales that are conceptually similar than with other scales (see Table 3). For 

example, the motivation scales (Academic Discipline, Commitment to School, and Optimism) 

showed generally larger correlations with each other (range r = .50 to .54, median = .53) than 

with other scales (range r = .28 to .63, median = .44). A similar pattern was seen for the self- 

regulation scales (Managing Feelings, Orderly Conduct, and Thinking before Acting) in which 

these scales tend to relate more strongly among themselves (range r = .49 to .62, median = .55) 
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than with other scales (range r = .25 to .55, median = .37). Although the social engagement 
scales (Family Attitude toward Education, Family Involvement, Relationships with School 
Personnel, and School Safety Climate) also showed a pattern of convergent/discriminant 
correlations, this pattern was less clear. Nevertheless, these scales still correlated more strongly 
among themselves (range r = .31 to .57, median = .45) than with other scales (range r = .25 to 
.63, median = .38). (See Table 3 for a full intercorrelation matrix of ENGAGE scales.) 
Relationships with Other Constructs 

ENGAGE scales have the expected relationships with other constructs as well. We 
examined correlations with behavioral indicators (e.g., absenteeism, times without homework), 
academic achievement (e.g., prior grades, EXPLORE scores), school-level factors (e.g., percent 
minority, average class size), and demographic factors (e.g., gender, parental education). The 
following sections detail these results. 

Behavioral indicators. As shown in Table 4, ENGAGE scales are mildly to moderately 
related to time spent on homework assignments (Hours Spent on Homework; range r = .13 to 
.24, median = .17). In addition, ENGAGE scales are moderately to strongly (negatively) related 
to frequency of students not having their homework completed (Homework Not Done range r = - 
.19 to -.58, median = -.30). Further ENGAGE scales are mildly to moderately (negatively) 
related to time spent watching television, playing videogames, or browsing the Internet for 
nonacademic purposes (Media Time range r = -.16 to -.28, median = -.19); times absent, tardy, 
and/or skipping class (Absenteeism range r = -.19 to -.36, median = -.22); and less strongly 
related to the number of times that a student has changed schools (Times Changed School range 
r = -.07 to -.14, median = -.10). Thus, ENGAGE scales are correlated with several key 
behavioral indicators that have been linked to academic success in the literature (Kaufman & 
Bradbury, 1992; Rumberger, 1995; Rumberger & Larson, 1998). 
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TABLE 4 

Correlations between ENGAGE Grades 6-9 Scales and Behavioral Indicators 


Behavioral Indicators 


Scales 

Hours 
Spent on 
Homework 

Homework 
Not Done 

Media 

Time 

Absenteeism 

Times 

Changed 

School 

Academic Discipline 

.24 

-.58 

-.27 

-.36 

-.14 

Commitment to School 

.17 

-.27 

-.18 

-.24 

-.08 

Family Attitude toward 
Education 

.14 

-.23 

-.16 

-.22 

-.07 

Family Involvement 

.20 

-.30 

-.19 

-.21 

-.10 

Managing Feelings 

.17 

-.34 

-.28 

-.26 

-.13 

Optimism 

.13 

-.30 

-.17 

-.20 

-.12 

Orderly Conduct 

.18 

-.36 

-.27 

-.33 

-.14 

Relationships with School 
Personnel 

.17 

-.28 

-.17 

-.21 

-.09 

School Safety Climate 

.13 

-.19 

-.17 

-.19 

-.08 

Thinking Before Acting 

.16 

-.33 

-.24 

-.20 

-.10 

Median 

.17 

-.30 

-.19 

-.22 

-.10 


Note. Range N = 2,741 to 4,619. Correlations with absolute value exceeding .05 are significant 

(P<- oi). 


Academic achievement. As shown in Table 5, ENGAGE scales are positively associated 
with indicators of academic achievement. For example, they are moderately related to school 
grades earned prior to students completing ENGAGE (range r = . 15 to .52, median = .30). Based 
on the literature, the associations of ENGAGE scales with standardized achievement measures, 
such as EXPLORE scores, are expected to be somewhat lower than those with grades. 
Measures of academic behavior tap achievement-related content that is expected, and has 
generally been shown, to be distinct from most cognitive skills and standardized achievement 
measures (e.g., Lounsbury, Sundstrom, Loveland, & Gibson, 2003; Robbins et ah, 2006; Watson 
& Clark, 1993). Table 5 also shows relationships between ENGAGE scales and EXPLORE 
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scores. As can be seen, these relationships are generally low to moderate and provide further 
evidence of discriminant validity. 


TABLE 5 

Correlations between ENGAGE Grades 6-9 Scales and Academic Achievement 


Academic Achievement 


Scales 

Prior 

EXPLORE 

EXPLORE 

EXPLORE 

EXPLORE 

EXPLORE 

Grades 

Composite a 

English a 

Math 3 

Reading a 

Science a 

Academic 
Discipline 
Commitment to 
School 

.52 

.27 

.25 

.22 

.22 

.26 

.31 

.23 

.21 

.20 

.19 

.20 

Family Attitude 
toward Education 

.27 

.22 

.20 

.18 

.19 

.19 

Family 

Involvement 

.28 

.13 

.13 

.08 

.12 

.13 

Managing Feelings 

.31 

.21 

.20 

.17 

.16 

.21 

Optimism 

.31 

.14 

.13 

.12 

.11 

.14 

Orderly Conduct 

.37 

.28 

.26 

.22 

.23 

.27 

Relationships with 
School Personnel 

.28 

.13 

.14 

.09 

.11 

.13 

School Safety 
Climate 

.15 

.08 

.08 

.06 

.05 

.09 

Thinking Before 
Acting 

.28 

.15 

.14 

.11 

.12 

.15 

Median 

.30 

.18 

.17 

.15 

.14 

.17 

Note. Range N = 2,936 to 4,537. 

a Range N = 

2,785 to 3,765. 

Correlations with absolute value exceeding 


.05 are significant (p < .01). 


School factors. As expected, ENGAGE scale scores are generally unrelated to school- 
level factors (see Table 6), including the percent of minority students in a school (range r = .00 to 
-.17, median = -.08), the percent of free or reduced-lunch recipients (range r = -.05 to -.19, 
median = -.11), average class size (range r = .01 to .10, median = .06), and student-teacher ratio 
(range r = .02 to .15, median = .08). These findings are consistent with the literature showing 
that academic behavior measures are generally unrelated to these types of school factors 
(Glovinsky-Fahsholtz, 1992; Wyss, Tai, & Sadler, 2007). 
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TABLE 6 

Correlations between ENGAGE Grades 6-9 Scales and School Factors 


Scales 


School-level Factors 


% minority 

% free 

lunch 

Average 
Class Size 

Student- 

Teacher 

Ratio 

Academic Discipline 

-.12 

-.14 

.08 

.08 

Commitment to School 

-.05 

-.11 

.05 

.10 

Family Attitude toward Education 

-.02 

-.09 

.06 

.08 

Family Involvement 

-.08 

-.10 

.02 

.03 

Managing Feelings 

-.10 

-.13 

.09 

.10 

Optimism 

-.04 

-.05 

.01 

.02 

Orderly Conduct 

-.12 

-.17 

.10 

.12 

Relationships with School Personnel 

-.07 

-.08 

.01 

.07 

School Safety Climate 

-.17 

-.19 

.03 

.15 

Thinking Before Acting 

.00 

-.06 

.08 

.03 

Median 

-.08 

-.11 

.06 

.08 


Note. Range N = 2,975 to 4,660. Correlations with absolute value exceeding .05 are significant 
(p<- 01). 


Demographic factors. Table 7 features relationships between ENGAGE scales and 
demographic factors. ENGAGE scales showed low to moderate relationships with demographic 
factors. Instances in which these relationships were moderate in magnitude (e.g., Orderly 
Conduct with male gender; Family Involvement with home resources; Commitment to School 
and Family Attitude toward Education with family educational aspirations) make intuitive sense 
and are consistent with previous research indicating that males are more likely to engage in 
disruptive behaviors (e.g., Cohn & Modecki, 2007; Zimmer-Gembeck, Geiger, & Crick, 2005), 
family resources are related to parental involvement in their children’s academic endeavors 
(Alexander et ah, 2001; Desimone, 1999; Sui-Chu & Willms, 1996), and educational 
achievement is related to family educational aspirations (Fan & Chen, 2001; Hill & Tyson, 
2009). Although relationships between academic achievement and academic behaviors may vary 
somewhat by demographic groups, this study did not address this issue. 
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TABLE 7 

Correlations between ENGAGE Grades 6-9 Scales and Demographic Factors 




Demographic Factors 


Scales 

White 

M . Parent Home 

1 Education Resources 

Fam. Edu. 
Aspirations 


Academic Discipline 

.13 

-.17 

.17 

.25 

.23 

Commitment to School 

.07 

-.16 

.16 

.22 

.30 

Family Attitude toward 
Education 

.06 

-.08 

.20 

.26 

.30 

Family Involvement 

.11 

-.05 

.22 

.30 

.21 

Managing Feelings 

.15 

-.20 

.15 

.18 

.14 

Optimism 

.05 

-.06 

.17 

.23 

.20 

Orderly Conduct 

.17 

-.25 

.13 

.18 

.14 

Relationships with School 
Personnel 

.11 

-.10 

.10 

.19 

.15 

School Safety Climate 

.12 

-.06 

.04 

.08 

.06 

Thinking Before Acting 

.04 

-.08 

.10 

.16 

.10 

Median 

.11 

-.09 

.16 

.21 

.18 

Note. Range N = 2,975 to 4,660. 

Correlations with absolute value exceeding .05 

are significant 


(P<-01). 


Predictive Validity 

As stated previously, ACT researchers partnered with the school districts that participated 
in the field study to follow the 4,660 participating students as they progress through high school. 
Given the difficulty in studying dropout before the age of 16 due to state mandatory school 
attendance requirements, we used early high school GPA (during either 9th or 10th grade) as a 
direct measure of academic performance. Failing one or more courses — and by extension having 
a low high school GPA — can be considered a proxy of eventual dropout risk (see Mac Iver, 
2010; Rumberger & Lim, 2008). 


23 


Analyses 

Regressions. Linear regression models were run to test our hypotheses that academic 
behaviors (i.e., psychosocial and behavioral variables) are predictive of high school GPA after 
controlling for the effects of traditional predictors (school-level, demographic, and academic 
achievement variables). In model 1, high school GPA was regressed on school and demographic 
characteristics. In model 2, measures of academic achievement were added to the model, 
including prior grades earned and EXPLORE Composite score. In model 3, the behavioral 
indicator variables were added followed by the psychosocial variables in model 4. Multiple R 
was used to measure the overall predictive strength of the four models. Fitting the models 
sequentially allows us to measure how much each set of predictors adds to the overall predictive 
strength. 

Capture rates. True positive and capture rates are measures that are meaningful if a 
model’s predicted values are used to identify students who are at high risk of failing 
academically in the future. Here, we define the true positive rate as the probability of an 
unsuccessful outcome, given that a student is flagged for intervention. The capture rate is the 
probability of a student having been flagged, given that they have an unsuccessful outcome. The 
true positive and capture rates can be calculated as a function of R for linear regression models 
(Allen, Robbins, & Sawyer, 2010). We assumed that students ranking in the bottom p% of 
predicted values (p = 5, 10, 25) would be flagged, and calculated true positive and capture rates 
for identifying students with early high school GPA less than 2.0. Three models for identifying 
at-risk students were compared for each outcome: (a) random selection; (b) prior grades and 
EXPLORE Composite scores; and (c) prior grades, EXPLORE Composite scores, psychosocial 
variables, and behavioral indicators. 

Dominance analysis. Because of correlation between and within sets of predictor 
variables, the relative importance of a set of predictor variables cannot be detennined from 
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regression coefficients or beta weights. Moreover, while the sequential regression models allow 
us to measure how much each set of predictors adds to the overall strength in predicting early 
high school GPA, they do not allow us to detennine the relative importance of each set of 
predictors. 

Accordingly, we used the dominance analysis technique (Azen & Budescu, 2003) to 
compare the relative importance of each set of predictors. Dominance analysis is based on an 
examination of the R 2 values for all possible subset models — or all possible models fonned by 
each possible combination of predictor variables. The R 2 attributed to each predictor is 
detennined by analyzing the R~ values for all subset models to which the predictor belongs. The 
total amount of variation explained by each set of predictor variables is then obtained by 
summing the R 2 attributed to each predictor within the set. The dominance analysis technique is 
attractive in that the importance of each predictor is estimated irrespective of order of model 
entry. It is also conceptually appealing in that it makes no assumptions about the causal 
relationships between predictor variables in the model. For example, the dominance analysis 
technique allows us to measure the importance of academic achievement measures and academic 
behaviors in predicting early high school GPA, without requiring us to specify a model for the 
relationship between the psychosocial and academic achievement variables. Such a model would 
likely need to capture complex relationships — such as the effects that motivation, social 
engagement, and self-regulation have on academic achievement and subsequent behavior. 

Moderation between achievement and psychosocial risk. Finally, we examined the 
combined effects of achievement and psychosocial risk by sorting our participants into a 3 x 3 
matrix of high-medium-low achievement (based on EXPLORE Composite scores) along one 
dimension and high-medium-low psychosocial scores (based on ENGAGE Grades 6-9) along the 
other dimension. In addition to examining main effects of these predictors, we explored the 
possibility of moderation between academic achievement and psychosocial scores. 
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Results 

Prediction of high school GPA. All academic, psychosocial, and behavioral indicators 
were significantly related (at the bivariate level) to early high school GPA. The bivariate 
associations of early high school GPA with middle school grades (r = .64) and EXPLORE 
Composite score ( r = .56) were the largest in magnitude. These were followed by the 
associations of the psychosocial factors (range r = .18 to .48; median = .28), behavioral 
indicators (range r =|.10| to |.41|; median = |.20|), demographic factors (range r = .05 to .29; 
median = .19), and finally school-level factors (range r = | .0 1 1 to |.21|; median = |.16|) (see Table 
8). This table also features the results of the four sequential linear regression models; beta 
weights for the variables are presented. The first model included school-level and demographic 
predictors and yielded an R of .397. The second model added academic achievement variables, 
and this model yielded an R of .7 1 1 . The third model added the behavioral indicators and yielded 
an R of .730; the fourth model added the psychosocial variables and yielded an R of .742. F- 
statistics confirmed that each sequential model’s increase in R was a statistically significant 
increase over the previous model. 
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TABLE 8 

Correlations and Beta Weights for Predictors of Early High School GPA 


Predictor 



Model 


V 

1 

2 

3 

4 

School-level 






% free lunch 

-.16 

-.049 

-.017 

.000 

.019 

% minority 

-.21 

-.047 

-.054* 

-.059* 

-.061* 

Average Class Size 
Demographic 

.01 

-.076* 

-.112* 

-.108* 

-.098* 

Gender (male) 
Race/Ethnicity 

-.17 

-.184* 

-.101* 

-.083* 

-.068* 

White 

.26 

.160* 

.063* 

.052* 

.036 

Black 

-.16 

-.061* 

-.016 

-.025 

-.026 

Hispanic 

-.17 

-.017 

-.014 

-.020 

-.023 

Asian 

.05 

.079* 

.030 

.025 

.019 

Parent’s Education 
Academic Achievement 

.29 

.211* 

.040* 

.030 

.028 

Prior Grades 

.64 


.431* 

.353* 

.316* 

EXPLORE Composite Score 
Behavioral Indicators 

.56 


.302* 

.296* 

.297* 

Number of Days Absent 

-.22 



-.050* 

-.041* 

Skipped Class 

-.22 



-.053* 

-.023 

Homework Not Done 

-.41 



-.146* 

-.102* 

Was Held Back 

-.20 



-.022 

-.020 

Media Time 

-.19 



-.007 

.011 

Psychosocial 






Academic Discipline 

.48 




.088* 

Managing Feelings 

.34 




.027 

Commitment to School 

.26 




-.054* 

Family Attitude toward Education 

.26 




.046* 

Family Involvement 

.28 




.028 

Optimism 

.27 




-.009 

Orderly Conduct 

.43 




.081* 

Relationships with School 
Personnel 

.28 




.005 

School Safety Climate 

.18 




.021 

Thinking Before Acting 

.24 




-.049* 

Model R 


.397 

.711 

.730 

.742 


Note, n = 3,278 for multiple regression models. Correlations with absolute value 
exceeding .05 are significant (p < .01). Predictors marked with an asterisk are 
significant (p < .01). 
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In Table 8, effect sizes are represented by both Pearson’s correlations (r) and beta 
weights ( b ). Both metrics can be interpreted as the estimated increase in GPA (in standard 
deviation units) associated with a standard deviation increase in the predictor. In the final model, 
prior grades ( b = .316) and EXPLORE Composite score (b = .297) remained the strongest 
predictors, with significant incremental validity added by Academic Discipline ( b = .088), 
Orderly Conduct {b = .081), Family Attitude ( b = .046), Homework Not Done {b = -.102), and 
Number of Days Absent ( b = -.041). Interestingly, the psychosocial variables Commitment to 
School ( b = -.054) and Thinking Before Acting ( b = -.049) had negative weights in the final 
model. This is likely a product of model multicollinearity and employing a large number of 
predictor variables (26 total) in model 4, as the bivariate correlations between Commitment to 
School and Thinking Before Acting with early high school GPA were positive and significant. 

Because students are nested within high schools, we considered using hierarchical linear 
modeling (HLM; Raudenbush & Bryk, 2002) to model the outcomes of interest. However, we 
chose to use ordinary least squares regression because it pennitted use of the dominance analysis 
technique (described earlier). To test the sensitivity of our model to school effects, we also 
modeled early high school GPA using hierarchical linear regression with random intercepts for 
each high school. Although there was evidence of variation across high schools in average high 
school GPA (the variance of intercepts across schools was estimated at 0.06, p < .05), the HLM 
model results were very similar to the multiple linear regression results. The regression 
coefficient estimates for the student-level predictors (demographics, academic achievement, 
psychosocial factors, and behavioral indicators) differed by 0.02 or less. All student-level 
variables that were significant in the multiple regression model were significant in the HLM 
model, and vice-versa. Thus, only the results of the regressions are reported in this document. 

Accuracy of classification. Table 9 features accuracy of classification models based on 
true positive and capture rates. Both rates depend on the percentage of students who are 
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flagged — true positive rates decrease as the percentage of students flagged increase, whereas 
capture rates increase with an increase in the percentage of students flagged. As shown in Table 
9, use of prior grades and achievement score results in substantial increases in accuracy (i.e., 
increased true positive rate) over random selection for identifying students who subsequently 
earned a low GPA (i.e. , <2 .0) during the 9th or 10th grades (probability increases over random 
selection by 0.609, 0.543, and 0.391 for 5, 10, and 25% of students flagged). Further, the 
combination of prior grades, EXPLORE Composite score, and psychosocial and behavioral 
indicators results in the highest level of accuracy (a 0.609 probability increase over random 
selection, for a total accuracy of 0.905 when 5% of students are flagged). The capture rates also 
demonstrate the incremental utility of using psychosocial and behavior measures. When 25% of 
students are flagged, 55.5% of the students who will struggle academically in high school are 
included among those flagged based on prior grades and EXPLORE Composite scores. This 
percentage increases to 58% when psychosocial and behavioral indicators are also used for 
flagging. 
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TABLE 9 

Accuracy of Identifying Students with Early High School GPA < 2.0 


Flagging Variables 

True Positive 
Rate 

Capture Rate 

5% flagged 



1. None (Students Flagged at Random) 

0.296 

0.050 

2. Prior Grades, EXPLORE Composite 

0.873 

0.147 

3. Prior Grades, EXPLORE Composite, 
psychosocial variables & behavioral 
indicators 

0.905 

0.153 

10% flagged 



1 . None (Students Flagged at Random) 

0.296 

0.100 

2. Prior Grades, EXPLORE Composite 

0.804 

0.271 

3. Prior Grades, EXPLORE Composite, 
psychosocial variables & behavioral 
indicators 

0.839 

0.283 

25% flagged 



1. None (Students Flagged at Random) 

0.296 

0.250 

2. Prior Grades, EXPLORE Composite 

0.658 

0.555 

3. Prior Grades, EXPLORE Composite, 
psychosocial variables & behavioral 
indicators 

0.687 

0.580 


Note. The true positive rate is the probability of having early high school GPA 
< 2.0 among students scoring in the bottom p% on the flagging variables (p = 5, 
10, 25). The capture rate is the probability of scoring in the bottom p% on the 
flagging variables among students with high school GPA < 2.0. 


Dominance analysis. Figure 1 reports the relative importance of each set of predictors in 
predicting early high school GPA using the dominance analysis technique (Azen & Budescu, 
2003). One of the primary benefits of using the dominance analysis technique is that the relative 
importance of sets of predictors are measured, irrespective of sequence of entry into the model 
and irrespective of theorized directions of causality between predictors. The results in Figure 1 
show that prior grades were the most important predictor, with 30% of the explained variation 
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attributed to it. Next, came EXPLORE Composite (25%), psychosocial factors (23%), behavioral 
indicators (10%), demographic variables (9%), and school characteristics (3%). Although the 
sequential regression models suggest that academic behaviors (i.e., psychosocial factors and 
behavioral indicators) add a significant but limited amount of explained variance to the model, 
the dominance analysis shows that academic behaviors are a key contributor (explaining 33% of 
variation in early high school GPA) when the sequence of model entry is ignored. 


FIGURE 1 Relative Strength of Predictors of Early High School GPA 


Academic 

Behaviors 


Psychosocial 

Factors 

23% 


Behavioral 

Indicators School Factors 

10% 3% Demographics 



EXPLORE 


Composite 

25% 


Interplay of academic achievement and behavior. Another way of presenting our study 
findings for early high school GPA is illustrated in Figure 2. To create this figure, students were 
classified into subgroups according to their EXPLORE Composite scores and their average 
ENGAGE scale scores. For each set of scores, students were classified into one of three groups: 
lower quartile (lowest 25%), middle quartiles (middle 50%), or upper quartile (highest 25%). On 
the left side of this figure, the first three bars show students who scored in the lowest quartile of 
EXPLORE Composite scores. Within this group, students who scored higher on ENGAGE 
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tended to earn higher GPAs. This pattern is consistent across EXPLORE score groups. Thus, 
Figure 2 illustrates the presence of two main effects on early high school GPA. The first main 
effect is that of prior academic achievement, as measured by the EXPLORE Composite score 
F( 2, 3422) = 198.9, p < .01. The second main effect is that of psychosocial factors, as measured 
by the average rank of the ten scales of ENGAGE F (2, 3422) = 112.2, p < .01. Further, we 
tested for moderation (i.e., the interaction in “cell” or column differences between academic 
achievement and psychosocial factors) and results showed that moderation was indeed present F 
(4, 3422) = 4.44, p < . 01. The test of moderation suggests that the strength of the effect of the 
psychosocial scores depends on prior academic achievement, or vice-versa. In particular, the 
differences between “low” and “medium” psychosocial scores increase as prior academic 
achievement level increases. This finding suggests that, although students of all achievement 
levels may benefit from positive academic behavior, students with higher academic achievement 


tend to benefit the most. 



32 


FIGURE 2 Average Early High School GPA by EXPLORE and Average ENGAGE Grades 6-9 
Score Groups 
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Summary and Discussion 

ENGAGE was developed to assess a range of academic behaviors believed to be related 
to academic performance and persistence based on existing theoretical perspectives (i.e., 
motivational, social engagement, and self-regulatory) and empirical evidence documented in the 
literature. The set of predictors used to model school success also included a range of more 
“traditional” variables, such as prior grades, standardized achievement test scores, demographic, 
and school-level factors. Altogether, the measurement model presented in this report is among 
the most comprehensive available in the literature. 

The field study that served as the final stage of instrument development focused on a 
large cohort of 7th- and 8th-grade students across 24 middle schools from 13 districts throughout 
the Midwest and Southern regions of the U.S. ACT researchers have been tracking these 
students through high school with a fairly high follow-up rate (71% of the original sample). 
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Based on the field study data, ENGAGE has good to excellent psychometric properties. 
Specifically, ENGAGE scales have good to excellent internal consistency reliability (median 
alpha = .87). The instrument showed a pattern of convergent/discriminant relationships with 
other variables and constructs consistent with the expectation for a measure of academic 
behaviors. For example, ENGAGE scales were more strongly correlated with prior grades than 
with achievement test scores. Further, they were strongly correlated with a variety of behavioral 
markers of academic success, such as regular homework completion and (lack of) absenteeism, 
and less correlated with school-level and demographic factors. Finally, ENGAGE demonstrated 
the expected higher-order structure (consisting of three higher-order factors: Motivation, Social 
Engagement, and Self-Regulation), which parallels that of the college version of ENGAGE 
(ACT, 2011b; Le et ah, 2005). 

This report expands previous ACT longitudinal research bridging high school and college 
(e.g., Robbins et al., 2006), which focused on the effects of achievement and academic behaviors 
on college academic performance and persistence (and culminated in the development of 
ENGAGE College). The current report focuses on the development of ENGAGE and its validity 
for predicting high school academic perfonnance, thus extending ACT’s academic behavior 
research to the transition from middle school to high school. The findings confirm that academic 
achievement indicators (i.e., prior grades, standardized achievement scores) are the strongest 
predictors of future academic success. These findings are consistent with those of earlier 
longitudinal studies, in which course perfonnance during middle school was a key indicator of 
subsequent academic performance and eventual high school graduation (e.g., Allensworth & 
Easton, 2005, 2007; Bowers, 2010; Mac Iver 2010). 

The findings also show that academic behaviors contribute to the prediction of future 
academic performance and thus can be useful in identifying middle school students who are at 
high risk of failing academically and dropping out of high school. This has significant 
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implications for combining academic behavior and achievement information to support the 
timely identification of at-risk students. As demonstrated in this study, the predictive factors in 
question were clearly present in middle school and can be assessed and used to help students to 
better prepare for — and successfully navigate — the transition from middle to high school. 

With this enhanced arsenal of information, intervention programs can be more 
successfully directed toward the students who need help. Compared to early warning systems 
based on archival data alone (e.g., Balfanz et ah, 2007), our findings show that including 
academic behaviors provides significant (though modest) incremental power for predicting later 
academic achievement. More important than the improvement in prediction, measuring 
academic behaviors can help educators understand why students are at risk. As the dominance 
analysis shows (33% of the explained variation in early high school GPA is attributed to the 
combination of psychosocial factors and behavioral indicators), these factors play a prominent 
role in understanding students’ risk for academic difficulties and will be key in intervening with 
students who are at risk. 

By measuring the range of characteristics related to academic success (or risk), educators 
can align interventions to students’ unique needs and have a better chance of improving their 
performance. Educators need to be able to address students’ academic behavior needs before 
they manifest in failing grades or dropout. There is some research on the positive effects of self- 
regulatory, motivational, and social skill training on students (e.g., Pintrich & DeGroot, 1990) 
and on workers (e.g., Lord, Diefendorff, Schmidt, & Hall, 2010). More research is needed to 
better map interventions onto students’ academic behavior needs and to understand how 
institutional and family contexts can either facilitate or impede the effectiveness of such 
interventions. 

From an individual student narrative perspective (cf. McAdams & Olson, 2010), 
understanding the unique interplay of school, family, and individual achievement and academic 
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behavior factors allows schools to better know each student, be able to tell that student’s “story,” 
and use the aforementioned factors to make more informed decisions (e.g., Balfanz et al., 2007). 
For example, in our interactions with school administrators, we have found that those students 
who are at moderate risk but who are not acting out (and thus not drawing as much attention to 
themselves) are often missed by school personnel trying to identify students who need additional 
behavior support or interventions. These students actually may be among the most responsive to 
intervention and prevention strategies. Thus, we see the use of measures of academic behavior 
as key to helping tell each student’s “story” and to tailoring interventions in order to increase a 
student’s chance of success, whether in core academic areas (e.g., math, English) or in the 
behaviors and attitudes that support academic performance (e.g., positive attitudes toward 
education, prosocial behavior, increased engagement, time management, etc.). 

At the institutional level, some of the ways that measures of academic behavior can be 
used include: (a) use in student-level data dashboards and other early warning systems to identify 
at-risk students, and (b) looking at aggregate data (at the classroom, school, or district level) to 
monitor student characteristics and plan system- wide interventions and resources. At the family 
level, measures of academic behavior can be used to (a) provide parents/guardians with 
information that can facilitate conversations and engagement regarding their students’ academic 
lives, and (b) help them monitor whether their students are developing the characteristics needed 
for academic success. 

Because ENGAGE measures ten different facets of academic behavior, in addition to the 
behavioral indicators included, it can provide educators with a broader range of information 
about the reasons students are at risk (i.e., their strengths and weaknesses). ENGAGE scores 
provide information that educators can use to better target interventions and provide the 
particular kinds of support students need most. This would allow educators to intervene based on 
students’ attitudes, perceptions, and behavior, and provide interventions designed to increase 
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positive attitudes and behaviors and decrease risk. To help educators take these next steps, user 
documentation for ENGAGE provides samples of interventions that can be used by educators to 
help students leverage their areas of strength and develop their areas of need (ACT, 2011c). In 
addition, ACT has developed a set of behaviorally-anchored rating scales (ENGAGE Teacher 
Edition; ACT, 2010) that can be used by educators to rate student behavior, track student 
progress over time, and determine next steps. These scales are designed to complement 
ENGAGE and assess the same three broad domains important for academic success (i.e., 
Motivation, Social Engagement, and Self-regulation). By aggregating the behavioral ratings 
across students (e.g., across those participating in an intervention or across an entire school), 
educators can assess the effectiveness of interventions, as well as their own progress in 
improving students’ academic behavior. Finally, ENGAGE is part of a suite of academic 
behavior assessments that measure important predictors of performance and persistence from 
middle school to college, as well as the workplace. 

Understanding student risk and student retention over time is essential if we are to reduce 
the shockingly high dropout rates and low achievement rates in our K-12 educational system (cf. 
Balfanz et ah, 2007; Rumberger & Lim, 2008). As the research findings presented in this report 
suggest, good academic behaviors, in combination with academic achievement, provide the 


foundation for future academic success. 
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